Measuring the Value of Big Data Exploitation Systems: Quantitative, Non-Subjective Metrics with the User as a Key Component

A big data visualization system is comprised of a diverse set of components. Obvious components include the data set, an analytic acting on the data set, and the visualization of the data set and/or results from the analytic, with one or more of each of these components in the system. The consumer, or user, of the visualization is a less obvious component of this system, but is nonetheless critical to understand in gauging system performance. There are well-defined, quantitative metrics to describe the data set (how big?, what kind of data?), the analytic (how quickly does it converge?), the visualization (how over-plotted?), and the IT infrastructure (how much memory?). However, the user operates within an additional set of constraints provided by her context —the performance of the supporting IT infrastructure, time constraints, distractions, etc. As such, understanding the user-in-context has typically employed subjective and qualitative approaches (did you enjoy the presentation?) rather than objective and quantitative as seen in the other measures of the system. Such approaches result in inappropriate holistic assessments of systems with some components measured quantitatively (and often in great detail) while others are measured qualitatively, altogether resulting in unforeseen potential risks in overall system performance. Draper Laboratory is developing quantitative, nonsubjective measures to describe system-level performance within and between composed big data systems. These measures leverage classical psychometrics, but extend to include dynamic behaviors and physiologic responses. We have previously developed a measure of Engagement, and are currently working to apply that, and related measures, to assess the performance of big data analysis systems.